I’ve spent a bit of time setting up some basic metric collection for this website. So here are a few notes on how I put things together.

As a general overview, I’m collecting metrics by instrumenting my webapp code with the Opentelemetry SDK. Those metrics are sent to an Opentelemetry collector. The collector then exports metrics to a Prometheus backend, and I’ve set up a Grafana instance for visualizing the metrics.

Each of the components in that diagram lives in a docker container, and I deploy everything with docker compose. Isn’t that a little complicated for a simple website like this? Couldn’t I just use Google Analytics or something similar? Yes and yes. But I don’t like the idea of passing off my data to a third party, and I wanted to learn more about these technologies, so that’s what I did. One big learning from this was that the whole process of integrating Opentelemetry with Prometheus feels a bit rough. If I were to do it again, I’d probably skip Opentelemetry and just use Prometheus and Grafana.

Instrumentation

Instrumenting my webapp code with the Opentelemetry SDK was mostly a matter of following the otel documentation, although there were a few caveats. Number one caveat was metric names. While these metrics are created in opentelemetry land, where a “.” is typically a separator between namespace levels, eg. http_requests.server_duration, these metrics are ultimately heading to Prometheus land where the lengua franca is promql and a “.” means $#@&. I worked around this by renaming my metrics like so:

 1func newMeterProvider(r *resource.Resource) (*metric.MeterProvider, error) {
 2	metricExporter, err := otlpmetrichttp.New(context.Background())
 3	if err != nil {
 4		return nil, err
 5	}
 6
 7	var view metric.View = func(i metric.Instrument) (metric.Stream, bool) {
 8		s := metric.Stream{
 9			Name:        strings.ReplaceAll(i.Name, ".", "_"),
10			Description: i.Description,
11			Unit:        i.Unit,
12		}
13		return s, true
14	}
15
16	meterProvider := metric.NewMeterProvider(
17		metric.WithResource(r),
18		metric.WithReader(metric.NewPeriodicReader(
19			metricExporter,
20			metric.WithInterval(30*time.Second))),
21		metric.WithView(view),
22	)
23	return meterProvider, nil
24}

Line #9 of the above is the key part to this. It’s a relatively easy fix, but it definitely slowed me down a bit trying to figure out how and where to implement the fix.

Otel Collector and Prometheus Backend

Another aspect of my setup worth noting is the use of the otelhttp exporter. Typically, Prometheus scrapes endpoints for metrics, and Opentelemetry provides a Prometheus Exporter for just that purpose. However, if , for example, you have a high availability setup with many instances of your app, you will end up with a lot of different metrics getting bottle-necked at that one scrape endpoint. The options for resolving this are either to use the Prometheus Remote-Write Exporter or to use the Otlphttp Exporter and the Prometheus native Otlp endpoint. I opted for the later because the only major difference seemed to be where the metric format conversion happens. Either the otel collector converts the metrics before sending them, or prometheus does it when they are received. Part of the appeal of using opentelemetry is it being vendor agnostic. I should be able to store my metrics anywhere with my otel collector. So I thought it would make sense to keep it that way and let Prometheus handle the conversion. Here’s another diagram that hopefully makes all this clearer:

Next steps

Overall, my observability setup is still very much a work in progress, but I’ve enabled anonymous read-only access on my Grafana dashboard here for anyone weirdly bored enough to take a look. I plan on adding more metrics, and I’d also like to set up some alerting for if and when any anomalies pop up.